Efficient Unified Demosaicing for Bayer and Non-Bayer Patterned Image Sensors

As the physical size of recent CMOS image sensors (CIS) gets smaller, the latest mobile cameras are adopting unique non-Bayer color filter array (CFA) patterns (e.g., Quad, Nona, QxQ), which consist of homogeneous color units with adjacent pixels. These non-Bayer sensors are superior to conventional Bayer CFA thanks to their changeable pixel-bin sizes for different light conditions but may introduce visual artifacts during demosaicing due to their inherent pixel pattern structures and sensor hardware characteristics. Previous demosaicing methods have primarily focused on Bayer CFA, necessitating distinct reconstruction methods for non-Bayer patterned CIS with various CFA modes under different lighting conditions. In this work, we propose an efficient unified demosaicing method that can be applied to both conventional Bayer RAW and various non-Bayer CFAs' RAW data in different operation modes. Our Knowledge Learning-based demosaicing model for Adaptive Patterns, namely KLAP, utilizes CFA-adaptive filters for only 1% key filters in the network for each CFA, but still manages to effectively demosaic all the CFAs, yielding comparable performance to the large-scale models. Furthermore, by employing meta-learning during inference (KLAP-M), our model is able to eliminate unknown sensor-generic artifacts in real RAW data, effectively bridging the gap between synthetic images and real sensor RAW. Our KLAP and KLAP-M methods achieved state-of-the-art demosaicing performance in both synthetic and real RAW data of Bayer and non-Bayer CFAs

Fully Quantized Always-on Face Detector Considering Mobile Image Sensors

Despite significant research on lightweight deep neural networks (DNNs) designed for edge devices, the current face detectors do not fully meet the requirements for "intelligent" CMOS image sensors (iCISs) integrated with embedded DNNs. These sensors are essential in various practical applications, such as energy-efficient mobile phones and surveillance systems with always-on capabilities. One noteworthy limitation is the absence of suitable face detectors for the always-on scenario, a crucial aspect of image sensor-level applications. These detectors must operate directly with sensor RAW data before the image signal processor (ISP) takes over. This gap poses a significant challenge in achieving optimal performance in such scenarios. Further research and development are necessary to bridge this gap and fully leverage the potential of iCIS applications. In this study, we aim to bridge the gap by exploring extremely low-bit lightweight face detectors, focusing on the always-on face detection scenario for mobile image sensor applications. To achieve this, our proposed model utilizes sensor-aware synthetic RAW inputs, simulating always-on face detection processed "before" the ISP chain. Our approach employs ternary (-1, 0, 1) weights for potential implementations in image sensors, resulting in a relatively simple network architecture with shallow layers and extremely low-bitwidth. Our method demonstrates reasonable face detection performance and excellent efficiency in simulation studies, offering promising possibilities for practical always-on face detectors in real-world applications.Comment: Accepted to ICCV 2023 Workshop on Low-Bit Quantized Neural Networks (LBQNN), Ora

Design of highly perceptible dual-resonance all-dielectric metasurface colorimetric sensor via deep neural networks

Colorimetric sensing, which provides effective detection of bio-molecular signals with one's naked eye, is an exceptionally promising sensing technique in that it enables convenient detection and simplification of entire sensing system. Though colorimetric sensors based on all-dielectric nanostructures have potential to exhibit distinct color variations enabling manageable detection due to their trivial intrinsic loss, there is crucial limitation that the sensitivity to environmental changes lags behind their plasmonic counterparts because of relatively small region of near field-analyte interaction of the dielectric Mie-type resonator. To overcome this challenge, we proposed all-dielectric metasurface colorimetric sensor which exhibits dual-resonance in the visible region. Thereafter, we confirmed with simulation that, in the elaborately designed dual-Lorentzian-type spectra, highly perceptible variations of structural color were manifested even in minute change of peripheral refractive index. In addition to verifying physical effectiveness of the superior colorimetric sensing performance appearing in the dual-resonance type sensor, by combining advanced optimization technique utilizing deep neural networks, we attempted to maximize sensing performance while obtaining dramatic improvement of design efficiency. Through well-trained deep neural network that accurately simulates the input target spectrum, we numerically verified that designed colorimetric sensor shows a remarkable sensing resolution distinguishable up to change of refractive index of 0.0086.N

PyNET-QxQ: A Distilled PyNET for QxQ Bayer Pattern Demosaicing in CMOS Image Sensor

The deep learning-based ISP models for mobile cameras produce high-quality images comparable to the professional DSLR camera. However, many of them are computationally expensive, which may not be appropriate for mobile environments. Also, the recent mobile cameras adopt non-Bayer CFAs (e.g., Quad Bayer, Nona Bayer, and QxQ Bayer) to improve image quality; however, most deep learning-based ISP models mainly focus on standard Bayer CFA. In this work, we propose PyNET-QxQ based on PyNET, a light-weighted ISP explicitly designed for the QxQ CFA pattern. The number of parameters of PyNET-QxQ is less than 2.5% of PyNET. We also introduce a novel knowledge distillation technique, progressive distillation, to train the compressed network effectively. Finally, experiments with QxQ images (obtained by an actual QxQ camera sensor, under development) demonstrate the outstanding performance of PyNET-QxQ despite significant parameter reductions.Comment: in revie

Novel Herbal Therapeutic YH23537 Improves Clinical Parameters in Ligature-Induced Periodontal Disease Model in Beagle Dogs

Currently, available medicine does not satisfy the clinical unmet needs of periodontal disease. Therefore, novel drugs with improved efficacy profiles are needed. We previously demonstrated that YH14642, water extracts of Notoginseng Radix and Rehmanniae Radix Preparata, improved probing depths in double-blind phase II clinical trial. However, it still has hurdles for commercialization due to the low efficiency of active compound extraction. To resolve this issue, we developed YH23537 through process optimization to extract active compounds efficiently while still achieving the chemical profile of YH14642. In this study, we investigated the therapeutic effects of YH23537 compared with YH14642 using a canine model of ligature-induced periodontitis. Human gingival fibroblast (hGF) cells were treated with various concentrations of YH23537 or YH14642 with lipopolysaccharide (LPS) for 24 hr. IL-6 and IL-8 levels in the conditioned media were determined using Luminex. Sixteen 3-year-old male beagle dogs had their teeth scaled and polished using a piezo-type ultrasonic scaler under general anesthesia and brushed once daily for the following 2 weeks. Two weeks after the scaling procedure, the left upper second premolar (PM2), third premolar (PM3), and fourth premolar (PM4) as well as the left lower PM3, PM4, and first molar (M1) were ligated with silk-wire twisted ligatures. The dogs were fed with soft moistened food to induce periodontitis for 8 weeks, and the ligatures were then removed. YH23537 and YH14642 were administered for 4 weeks, and clinical periodontal parameters such as plaque index (PI), gingival index (GI), probing depth (PD), clinical attachment level (CAL), and bleeding on probing (BoP) were determined before and 1, 2, 3, and 4 weeks after treatment. YH23537 inhibited IL-6 and IL-8 secretion in a dose-dependent manner in hGF cells stimulated with LPS. The IC50 values for YH23537 were 43 and 54 μg/ml for IL-6 and IL-8, respectively, while the values for YH14642 were 104 and 117 μg/ml, respectively. In the animal study, clinical parameters including GI, PD, CAL, and BoP were significantly increased after 8 weeks of ligature-induced periodontitis. The YH23537 300 and YH23537 900 mg groups had significant improvements in CAL from 1 to 4 weeks after treatment in comparison to the placebo group. GR values in the YH23537 900 mg group were decreased throughout the treatment period. GI values were also reduced significantly after 4-week treatment with 300 and 900 mg of YH23537. YH23537 at 300 mg doses showed comparable efficacy for CAL and GR with 1,000 mg of YH14642. YH23537 showed therapeutic efficacy against periodontitis in dogs, mediated by anti-inflammatory effects. These findings indicate that YH23537 has the potential for further development as a new drug for patients suffering from periodontal disease

Requirements for Upper-Limb Rehabilitation with FES and Exoskeleton

In the last work, we have presented the scope of our project, i.e. use cases of activities of daily living (ADL) for the on-going project a.k.a. iCARE. The project mainly handles the upper-limb rehabilitation in general, however, we have narrowed down the scope and focus on, in terms of the phase of the stroke recovery, the target body area of rehabilitation and the level of muscle function. In this paper, we have drawn the user and system requirements before design the specific functions of the targeted device. First, we have defined the stakeholders for the device and the rehabilitation service scenarios. Next, the user requirements are defined and finally the related system requirements are drawn.This research was financially supported by the Ministry of Trade, Industry and Energy (MOTIE) and Korea Institute for Advancement of Technology (KIAT) through the International Cooperative R&D program (P0007114-InterConnected Intelligent Sensing and Actuation Solutions for At-home Rehabilitation (iCARE)(2020))